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traffic and determining which traffic 
does not belong in a network are 
disclosed. Intrusion detection is 
performed over a period of time, 
looking for behavioral patterns within 
networks or information systems 
and generating alerts when these 
patterns change. The intrusion 
detection system intelligently forms 
correlations between disparate 
sources to find traffic anomalies. Over 
time, behaviors are predictive, and the 
intrusion detection system attempts to 
predict outcomes, becoming proactive 
instead of just reactive. Intrusions 
occur throughout whole information 
systems, including both network 
infrastructure and application servers. 
By treating the information system 
as a whole and performing intrusion 
detection across .it, the chances of 
detection are increased significantly. 
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ADAPTIVE BEHAVIORAL INTRUSION 




This application claims the benefit of U.S. Provisional Patent Application No. 
60/368,629, filed March 29, 2002, entitled "Adaptive Behavior Intrusion Detection Systems 
and Methods," the entire contents of which are hereby incorporated by reference. 

Field of the Invention 
The present invention relates generally to methods and systems for providing security 
on a communication system and, more particularly, the invention relates to adaptive 
behavioral intrusion detection. 



Background of the Invention 
With the rise of the Internet and the use of computer networks by many businesses, 
network security has become increasingly important The rise of e-commerce has led to 
organizations opening their networks to wider audiences over the Intemet in order to stay 
15 conqjetitive. Such open networks expose the organizations to intrusions— attempts to 
comprise the confidentiaHty, integrity, or availability, or to bypass the security mechanisms 
of a computer system or network. Additionally, companies storing vast amounts of consumer 
data need to provide some reasonable method for assuring privacy. 

Attackers or hackers have continued to alter their attacks and network subversion 
20 mefliods, and vulnerabiHties continue to exist in many areas including network 
misconfiguration, poorly engineered software, user neglect and carelessness, and basic design 
flaws in protocols and operating systems. Furthermore, as the sophistication of tools used by 
hackers has increased, the technical knowledge required to attack a network has fallen. 
Additionally, attacks are often the result of malicious insider activity which cannot be 
25 prevented by perimeter defenses. 

Intrusion detection is the process of monitoring the events occurring in a computer 
system or network and analyzing them for signs of intrusion. An intrusion detection system 
(IDS) is a software product or hardware device that automates the intrusion detection process, 
and an IDS typically includes three functional components: information sources, analysis, 
30 and response. Analysis strategy fells into two basic types: knowledge-based misuse 
detection and behavioral-based anomaly detection. Behavioral-based detection methods use 
information about repetitive and usual behavior on the systems fliey monitor and note events 
that divCTge from expected usage patterns. 
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tattmion detection aUows orgamzato » p««ct thet systems ftom fte fl^eate that 
come wia increasing newotkoomiectiviv and reUa«=e on infoima«» systems. Given to 

level and na«:e of modem ne^vork secrtiy teeaK, IDSs have gained acceptence as a 
„^ addition .0 every organization's seonrity inftastt-ctoe, IDSs au^maticdly revrew 
5 massive »nonn.s of ne^ork and system daia in «al time, identic snspioio.. aoti^ty. 
provide «»I-4ne animated notification te security personnel. guid« fW«r mvestr^tion. 
and sometimes antomatically respond «, specified attaclcs. Prop«ly ns«i, an IDS can d«eC 
«™n>on attacks, attempts ,o e:^loi. kno™r weaknesses, netivo* i^bes. or critical re«.»ro. 
overloads in a reasonably timely manner. By idemiiymg snccessfiU invriid activiry, IDSs can 
,0 indirecfly spotiigh. network and system vulnerabilities, en^ling fix« '^f^' 

comprehensive network security requires multiple layers. An effective IDS mdudes 
bofl. knowledge-based and behavioral-based components. Most vendors provide ne«.ork 
security products that protect only agams. known or "signatme" patterns of attack. «>d 
t^calbehavioral-basedcomponents are timitedte smgleanomafe, detection witiKnrtlookmg 
15 fer behavioral patt^s over a longer period of time. Existing products ignore tiooblesome 
^ behavioral patterns that have yet to be detected or documented. Hackers often foHow 
^behavioral patterns that double as caUing canis for their personal invasive teohn.^^ 
For example, a h«*=r may attack aU of the hacker's targeted netwoAs by a recogmz^le and 
s^pence of port access attempts, but tire patten, is recognized as odd or a^^mng 
20 »ly after an attack has occurred and a profile for ti»t behavior is d«umca^ and 
publicized. Signatine and basic behavioral me4»ls of flrreat — ^ ^ 
ley Ml short as hackers detennine new ways te attack or a^ust toir old behavror tt> attract 

less attentioii. . - . 

Many serious inttuders perform considerable amounts of probing woric wrttm. 
25 network tt.le.m how it is consttucted and mKlerstand its weaknesses prior to a conc^ted 

attack. Ms ,ecom»riss»«=e work is commonly recorded in automated server and netwo* 
logs. bu. inrgely remains unnoticed by most network IDSs if tire traffic anomaly does no. fit 
ti>e profiles of known or common -signati^e" hacks. Accordingly, there is a need tor a 

^ wHh adaptive technology ti^t, over time, gatirers formation on a particular syst^ 
30 and establishes, pattern of normal tiaffic. Such a system is able tt. more mte« 

detennine which network ttaffic signateres do not fit tire normal profiles for ti» mdrvrA^a 

system and alerti, an intrusion detection team for further investigation and approprrate raprd 

defensive action. 
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Summary' of the Invention 
Systems and methods for analyzing historical network traffic and determining which 
traffic does not belong in a network are disclosed. Intrusion detection is performed over a 
5 period of time, looking for behavioral patterns within networks or information systems and 
generating alerts when these patterns change. Normal traffic behavior is collected on a 
continuing basis to estabUsh a baseline for comparison with future network traffic. Once a 
statistically significant sample of historical data has been compiled, a behavioral intrusion 
detection agent is activated. The intrusion detection system intelHgently forms correlations 
10 between disparate sources to find traffic anomalies. Over time, behaviors are predictive, and 
the intrusion detection system attempts to predict outcomes, becoming proactive instead of 
just reactive. Intrusions occur throughout whole information systems, including bofli 
network infrastructure and appUcation servers. By treating the information system as a whole 
and performing intrusion detection across it, the chances of detection are increased 
15 significantly. 

An exemplary embodiment of a method according to the present invention for 
detecting network intrusion attempts associated with network objects on a communications 
network includes collecting normal tiraffic behavior associated with network objects on tiie 
network on a continuing basis to establish historical data regarding traffic across the network. 

20 Network fa^c associated with network objects on the network is monitored to detect 
anomalies, which are analyzed using the historical data. Alerts are generated identifying 
possible intiusion attempts based on analysis of the anomaUes. The historical data is 
continually updated based on the anomalies, the alerts, and network traffic. 

In an «cemplary embodiment, monitoring network traffic to detect anomalies may 

25 include monitoring network traffic for known strings and series of bytes tiiat indicate 
signature attacks. Monitoring network traffic may also include applying a series of rules to 
identify anomalous packets and adding the anomalous packets to an anomaly pool. In an 
exemplary embodiment, analyzing the anomaUes using the historical data includes analyzing 
packets in the anomaly pool independently of any of the series of rules that identified tiie 

30 packet for addition to the anomaly pool. Analyzing the anomahes using the historical data 
may also include conducting a threshold analysis to determine wheliier a data point is within 
threshold values. 

In an exemplary embodiment, generating alerts identifymg possible intrusion attempts 
includes adding alerts to an alert pool and releasing alerts Id a console for viewing by an 
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„pen.,or. According ,o softer ccmplary cmlxxltaeo.. E^-erating alerts may indad. an 
L system, wtee a ta. se. of alerts a« added an aler. pool. iBteme. 

addresses associated widr each alert ir. the alert pool are ,esolv«l «i.h a r^e and each a^ert 
in ,be aler, pool is renamed with a r^nre recognizable by an operator, a set of nrles rs apphed 
5 ^ selec, alerts ftonr the alert pool to be displayed on a console for viewing by W«>r. 
to rules comprising bigh level selection parameters that have been previously defined, and 
fte selected alerts are released by name to the console for viewing by «k operator. 

certain e=cempl«y embodiments of metods of fte present invention may l>e 
porfonned across a pluraUty otnetworl. and witir the results compiled in a global dab>b.se, 
10 andHstoricaldatamaybeupdatedbasedonresnltsindreglobaldataba^^ 

A computer storage medium storing a computer program, when «teoa.«J by 
computer^troUed apparatus, may cause dre computer-conttoUed apparau. » p«fc™ 
certainexemplaryembodim^tsofmethodsaccordingtothepresentinvention. Addrtionally. 

a comp^er-controlled apparatus may be operative for implementing certain exemplary 
15 embodiments ofmelhodsofthe present invention. 

to an exemplary embodiment of a system according «, the present mvention. an 
inwsion detection system for deleting netwo* intrusion attempts assocU«d wid. netw^ 
on a commumcations network includes a sensor conneoted ,„ the network fi=r 
monitoring ne«vork traffic associated with network objects on the network. Ihe sensor m^ 
20 inctade a knowledge-based component tor ex^nining network traffic fcr known stm^ and 
aeries of hytes that indicate signatme attacks and a packet logger for reading pack«. .n 
^k traao. classifring packets by protocols, and creating packages of compreased 
packets. A server competed to the sensor accepts real-time alerts for possrbl. srgpatirre 
.«„>ks. and a converter is provided for converting alerts ftom native signamre foomrt *, a 
25 unified fcnna. for storage in a. leas, one reMonal database. An andysis server r^s 
e,.^ packets flom dre sensor at periodic intervals, and the analysis server conducts a 
behavioral analysis of to data received ftom to sensor, lire a, least one «lationaI database 
stores raw packet data, behavioral data, and index data. 

Certain ^cemplary embodhnents of systems of to present invention may mctade a 
30 plumli., of sensors connected to to network Also, two or more virhral private network 
.mmels connecting to sensor to to network may be pmvided m c«.am exemplary 
embodiments. 
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Brief Description of the Drawings 
Figure 1 is an exemplary environment for operation of systems and methods 
according to the present invention. 

Figure 2 shows an overview of process flow of an embodiment according to systems 

5 and methods of the present invention. 

Figure 3 shows process flow for an embodiment of resource tracking and rules-based 
anomaly detection according to systems and methods of the present invention. 

Figure 4 shows process flow for an embodiment of anomaly pool analysis according 
to systems and methods of the present invention. 
10 Figure 5 shows process flow for an embodiment of alert classification according to 

systems and methods of the present invention. 

Figure 6 shows process flow for an embodiment of statistical generation according to 
systems and methods of the present invention. 

Figure 7 shows process flow for an embodiment of threshold analysis according to 
1 5 systems and methods of the present invention. 

Figure 8 shows process flow for an embodiment of alert correlation according to 
systems and methods of the present invention. 

Figure 9 shows process flow for an embodiment of global analysis according to 
systems and methods of the present invention. 
20 Figure 10 shows process flow for an embodiment of alert release system according to 

systems and methods of the present invention. 

DetaHed Description of the Invention 
In describing embodiments according to sj^stems and methods of the present 
25 invention, the following terms are used herein in the manner indicated below: 

Agent An IDS component that gathers system data, monitors system activity, and 
issues alerts upon detection of an intrusion. 

Alert A message sent by an IDS warning of a suspected or actual intrusion and 
usually calling for some sort of action in response. Also referred to as a notification. 
30 Alert Generation: Addition of an alert to a database to record a detected event AU 

generated alerts are not necessarily displayed or released to an operator of the IDS, but 
generated alerts remain in the database. 

Alert Release: Displaying an alert on a console for viewing by an operator of the 

IDS. 
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console: Anad.M.«a«veor— en.con^=..of«.n.S. O-^.^^-^ 
^ in»*ce (GUI) *.ough wMe. an opera., o, use, con.o,s ope,a«o»s ot«>e n)S and 

^"■^ rr r::. - con^a. ^ 

5 raw packet 

F«K;alation- An alert is made a high priority. 

^onls^^c: Asin..e«,».ofpac..s.«». A>so ^ » as fi-s. 

s..is«c: ^ eon. a ,eU«onsMp «ween ™^.e sen«a«on I 

"'^oene.^onms^: A n.ea.« or a .^e. aa. poin. of a (i- 
^„™ohorhowo««aas««iec^».sinagiven*neftan«). Also «fcned «. as 

^"^o. A^o.a«ono,asya.e»se^^poucyb,annna„*o.izedoajsiae,o^ 

oertam sys^n. wWnn to nemofc accessmg cerum mes. or « 
•^^ion detecSon sys^n. (IDS,: Ar, a„.on..ed sys^m *a. can ae.«. a secnri.y 
^""™^rr:tane.o.an..ne,...an.se^^o. 

Prediction: A calculated value ftatxs a guess of a future data point. 
■^jlTe!^ Aran^orad^^seaasens. — Any^S^ 

"^Protocol- A so. of forn^ aescribing how » «ansn.. da«, espedaUy aCK>sa a 

» is tlow level pro»col defining bi.- and b,«.rdering and fl» «ansnnss.on. 

IT Z Z^of *e bi. s^. Exa^ple^ are IP (in^me. proW). IPSeo. 

r mZol) TCP (— sion con^ol protocol), UDP (user da^ 
30 (secure mtemet protocol), ll-r (nam ^solution protocol), and 

protocol), ICMP rm«»« """"""^S' P"'^"' ' 
ofliers wUch are wen known to (hose sMed in fte ait 
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Raw data: Actual packet headers up through layer 4 (64-bytes) stored in a ffle. Raw 
data is captured on a sensor and transferred to servers for import into a database. Once 
transferred to the servers, the raw data is no longer necessary. 

Raw packet: Data pulled from a network packet and stored in a database in a field by 

5 field schema. 

Resistance to Change (RTC): The resistance of any tracked data point to being 
changed by an analysis routine (i.e., the data point's predictability). A data point with a high 
predictability resists change because it has been judged correct many times before. A data 
point with a low RTC is typically unpredictable because it has a high rate of change. 
10 Resource: A usable function of a network device, including servers, protocols, and 

services. 

Rule: A set of selection criteria used to select raw packets. 

Score (SCR): A human rating of any tracked data point Similar to strength but 
carries a heavier weighting when determining how normal a data point is. A small change in 
15 score is the equivalent of a large change in strength. This value can only be changed 
manually by an operator of the IDS. 

Sensitivity: A global adjust that may be used to make a sensor more or less sensitive 
to certain types of alerts, such as resource, signature, threshold, and the Uke. Sensitivity can 
be used to tune a sensor to bettrar fit a particular network. 
20 Sensor: A device placed on a host network to monitor. A sensor may perform two 

fimctions: knowledge-based intrusion detection and packet logging of behavioral packages. 
A sensor collects raw packet data and behavioral data necessary to perform any analysis. 
This includes alerts, raw packets, thresholds, firequencies, attackers, and the like. The data 
collected remains separate for each sensor, but has the same format and schema. 
25 Server: A network device having an address and transferring data on a network. 

Service: A high-level protocol that specifies a function that is carried by a low level 
protocol. An example is HTTP (hyper text transfer protocol) carried by TCP, DNS (domain 
name server) carried by UDP, or an Echo Reply ICMP packet 

Strength (STR): An IDS rating of any tracked data point that judges the data point's 
30 normalcy to the network. Typically, low values indicate normal and high values indicate 
abnormal. 

Strength-Score Value (SSV): A calculated value that indicates the severity of an alert 
regarding its danger to flie host network. Every alert has an SSV fliat indicates the threat 
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asse^^o^en. of ^ of to alett value is calculaW ustag fl,e s«n^ and ^ of 
fl„al=rtand«,ea,«fs.ou^. TVpioa..,. higher ,h. SSV, *e higher the fc^t 

•n^old: A tadce. of a da« c«ve .ha, brackets both above and below the data 
points, mdicating the eurve's v^-. A threshold contains four data 

L point on tracked curve. Two points are a close es.in.te of the curve (a .hut brac^ 

»a, I usnally wiftin 3-5% of the normal data point) and a second more conserv«.v. 

estinate aiat is a wid.rbn»=ket (usually «i«. 6-10% of the nonual data pomt). 

Systen. and me«>ods according to to present invention provide the abihty to analyze 
ii^calnetworkt^ffic anddetenninewUch traffic does no.belonginanetwo,tlmrus.on 

detecd^r is pertonned over a period of fae. instead of exanmnug infonnation <,».cldy and 
never again se«ng the infcnnation. Sys»ms and n.e«,ods of the present invenuon look for 
behavioral pa«ems within networks or hrfom>a.ion systems and generate alerts when these 
patterns change. Nonnal traffic behavior is collected on a couhnuing bas. to cstabh^ a 
Lelinefor comparison wimfhnnenetwcktraffic. Once a statistically signrfican. ^leof 
; u^cal data has been compilea, a behavioral intrusion detection agent rs achvated The 
intrusion detection system mtelligentiy forms correlations between disparate sources to find 
^ anomalies. Over time, behaviors are predictive, and the intiusion detection system 
a^ts to predict outcomes, becoming pn»ctive instead of Just reactive. Intmsrons o»cur 
^^ou.v*ole infbnnation systems, ir^luding bo* network inftastracture and apphcaUon 
0 servers. By treating the information syst«. as a whole and performing intiusron detec^n 
across it. fte chances of detection are increased significantty. 

The analysis of netwoflc-wide events is difficult because ti.e data comes ftom very 
«arsourc.sandanalyzingthcsourceda,.is challenging. Any deviation in fl« nonnal 
l^^or of fte infimnation system is a sign of possible inu^sion. The data comes ftom 
« sources including servers, network senses, and firewalls, not iust a single so.^. 

source data (e.g.. raw packet da..) is stored and analyzed over a flrs^ specified 
pliod of time, and behavioral events (..^, alerts) are stored and used for analy^s for a 
second, longer specified period of time. One e«mple is a one monti> tirstperiod of time and 
a one year second period of time. It should be unders«>od tiu. each of the time penods may 
30 bevariedaccordingtofteprefbrencesoftolDSusersandoperators. 

sources of .Backs are stored long-term in a historical database along w,«> an mdrcator 
otflreirhostiBtytofl^hostnetworic. -nds allows aU further anomalies and alerts to be 
^.ed to indicate tita. »e atiack was no. a single, isolated event but a repeat ftom a kno™ 
attacker. Additional atiacks. m unn. mise the indicator leadmg to fester «».lation. whde 
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periods of inactivity from the attacker lower fee indicator. Attackers are removed from the 
historic database only when they have decayed to point where they have reached a negative 
indicator below a predetermined value. This value is a software setting, ffigher values track 
attackers longer and lower values release them firom the database sooner. 
5 Additional features of an EDS according to this invention may include: analysis of bit- 

level packet detail; correlation of packet anomalies against IP-independent attack profiles; 
easy adaptabihty Aat includes signature intrusion detection systems; time based analysis that 
correlates profiles against packet anomaUes; port access attempts correlated by bolii time and 
sequence; and traffic data stored in and processed from a relational database (RDB). 
10 An exemplary environment for operation of systems and methods according to flie 

present invention is shown in Fig. 1 and includes a master server (also referred to as master 
ESP server), at least one real-time server, at least one converter, at least one relational 
database (RDB), at least one appUcation and analysis server, a web server, a session server, 
and a sensor (also referred to as an ESP sensor). Exemplary fimctionaUties of each of these 
1 5 components are discussed below. 

A master server serves an ESP database 104. Among other things, the master server 
keeps track of the job Ust, including pending, completed, and in-process jobs, and handles 
cooperative process-locking. The master server may also distribute sensors over RDB 
servers, stores global configuration values, stores sensor master variables (static configuration 
20 values for each sensor), and stores user accounts. In the embodiment shown in Figure 1, a 
session server 126 acts as the master server. It is well understood by those skilled in the art 
that any appropriate server may be the master server. Servers such Oracle or MySQL servers 
or other standard hardware running Linux/Unix based operating systems may be used for the 
master server, as well understood by those skilled in the art. 
25 As shown in Figure 1, real-time servers 106 and 108 accept sensor connections for 

real-time alerts. Real-time servers 106 and 108 maintain the sensor real-time tunnel (network 
connectivity), store real-time data in native format (native to signature system: Dragon, Snort, 
etc.), store logged data fiwm events (actual data capture at time of event), rebuild data 
sessions for appUcation servers, and may also act as converters. Real time servers are 
30 standard servers, well known to those skilled in the art, running LinuxAJnix based operating 
systems. Typically, Aey may include a dual-processor based system with a RAID array disk 
unit and a network file system, such as CODA, CIFS, or KFS, if the disk storage is not 
already netwoik-based, all of which is well understood by those sldlled in the art 
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A convener 110 reads re.l-Sme alerts bom native fonnaBed files, connects to a 
sensor's RDB (such as RDBs 112. 1 14. and 1 16). converts alerts ftom native sigoan^e forn^t 
U, unified fonnat, and may combine mnltiplc sensors in«> a single ESP sensor. Convert.. 110 
is standard hardware, well toown to those skilled in art. running UnnxrtJnnc based 
5 operating systems that is able to mo«rt network shared ffle systems and appropnate chent 
slftvJto^onnecttotheKDBs. Relational databases 1 12. 1 14. and ,16 (EBBs) sU« raw 
packet data, store behavioral data, and index data. KDBs 112. 114. and 116 send server^^ 
I web servers (such as web server 124). appUcation servers (such as appUcahon server 120). 
and analysis servers (such as analysis server ,22). RDBs 112. 114. and 116 aec^ da^fiom 
,0 converter 1 10. Additional storage, such as storage 1 18. may also be provided for ^ 
ftonr RDBS ,12. 114. and 116 or database ,04. RDBs 112, 114. and 1,6 may run on 
,.3ndard SQL-based database servers, such as *ose ftom Oracle or MySQL, whach « w.n 
„„a.^ by those sH„ed in fte art The RDBs may typically have 1 GB of RAM p» 
processor, and some configurations may include a storage area network (SAK) or network 
15 attached storage (HAS) for central database storage. 

An appHcadon server or servers 120 s«»es to sysKm's backend user mterftce code, 
serves appUcation data «. web servers (such as web server 124) and oflaer user interS^ 
systems. AppUcation s«ver 120 may serve any sensor. An analysis server or serves 122 
loolcs for pending jobs, locks jobs, marks jobs in-process, perfom^ analysis jobs, s,^ 
20 results (update behavioral profile), mad. jobs p„>cessed. and unlocks jobs. Ap^U^ 
serv^ 120 and server 122 may be combined in a single server. Apphcanon and 

analysis servers inctade standard hardware nmning a Unux/Onix based operating s>^ 
wifl. appropriate cUent softw... to comrec. to ti.e RDB servers, as weU m,ders«,od by tho» 
sldlled in fl» art open SSH, wUch is wen undersK>od by ti.ose skilled in .he art. may be 
25 nsed if encrypted communication, an, re.piredfixunfl« appUcation and analysUse^^ 

any web serves. 

A web sen«r ,24 processes w* «q»esls. caUs application server applets to produce 
appUcation data, and h«.aes us« in^rfcces. Web server 124 may no. access sensor date. A 
web server is s«ndanl hardware. weU known to those skiUed in d>e art. and typically ru^ on 
30 a LinnxAto based operating sys^mwi* appropriate cUentsoflware to com«c.«. RDB 

servers. Session server 126 handles web server session keys, stores session data, deletes old 
sessions (stale), and handles account. As noted above, session server 126 may. m some 
instances, be the master server. Standard s«ver hardware may be used for the session server. 
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such some of the servers described above, as is weU undeistood by those skiUed in flie art. A 
separate relational database 128 connected to web server 126 may be provided. 

A sensor (or ESP sensor) 102 includes tluree separate subsystems designed to perform 
three functions: signature checking, data capture, and security. Each sensor is a bastion host 
containing an internal firewall, a knowledge-based sensor, and a packet logger. The 
knowledge-based sensor is designed to handle signature detection only, and leaves the 
anomaly detection to the IDS analysis servers. The packet logger has complex filtering 
capabilities that can be used to capture only the desired data, but this is rarely employed. The 
more data the system captures, the better the analysis. Typically, sensor 102 includes a single 
or dual processors (for larger networks) and two network interface cards. Standard hardware 
runs Linux 2.4+ with the Linux socket filter, sock packet and ipchains features enabled. A 
virtual- private network tunnel or other transport tunnel is necessary to move data from the 
sensor to a server. As noted above, OpenSSH may be used to transfer behavioral packages to 
conversion and import servers. 

Sensors are designed as bastion hosts because they are employed in perimeters in 
front of any security devices. Typically, the sensors operate with a read-only wire for the 
sensing interfece, but a second network intetfece handles communications with the console, 
and as a security device, this interfece must be well protected. The internal firewall is used to 
protect the communications interfece and to protect the sensor for denial of service attacks 
using traffic shaping techniques. All packets on the communications interfece are fiilly 
defiagmented before being passed to the remainder of the IDS. 

As noted, sensor 102 includes a packet logger. On very high-speed networks, any 
analysis on the sensor hinders performance of the sensor and can lead to dropped packets. 
For ttiese reasons, the behavioral analysis occurs offline. The packet logger creates 
"packages" of compressed and encrypted traffic fliat is sent back to the analysis station at 
specified intervals, for example SO-minute or 60-nmiute intervals. Each package is 
unencrypted, decompressed, and fed into the behavioral analysis portion of the IDS. as 
further described below. 

The packet logger reads packets fix>m the networic interfece, classifies the packets by 
protocol, compresses the packets, and writes them to a disk using a double-buffered, threaded 
process. The analysis relies mainly on traffic patterns of a network, so the data is not 
necessary. Though there are reasons to log flxe data, it is not essential for behavioral analysis. 
On very hi^-speed networks, the sensor can be split into two parts, with one performing 
signature checking with a knowledge-based sensor, and the oflier performing packet loggmg. 

11 
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series of bytes that indicate attacks. Alerts appear on the console in real-time. The alerts are 
automatically prioritized by information contained about the source address in the behavioral 
database, whereas most signature-based processes prioritize alerts by ^e (alert name) only. 
The signature-based alerts are escalated from their default priority as necessary based on the 
5 source's strength, score, and the signature sensitivity setting for the sensor. 

Behavioral analysis includes block 300 through block 800. La blocks 300 through 
800, behavioral analysis is performed for an individual host information system (a collection 
of related sensors). Finally, global analysis occurs at block 900 and includes behavioral 
intrusion detection over sensors across multiple information systems in order to help find 

10 hackers across the Internet This can identify scanning soxirces and coordinated attacks. 

Behavioral analysis includes adaptive rules-based profiling using model-based 
reasoning. Behavioral patterns are abstracted by describing them as sequences and thresholds 
stored as historic rules. Current behavioral patterns are then checked against the historically 
predicted patterns and deviations are noted. Some deviations can be classified at the abstract 

15 level, while others require the IDS to find the associated source data and perform rules-based 
source analysis. Behavioral analysis uses information in the RDBs with the goal of detecting 
all intrusions. The sensor is connected to the remainder of the IDS and transfers the logged 
packets to the analysis server. Each packet is decompressed and imported into the RDB for 
the particular sensor, and analysis then begins. Since analysis is rule-based, once the data is 

20 ready, processing can be spUt across multiple servers allowing analysis to occur on clusters. 

As shown in Figure 2, behavioral analysis includes numerous steps. Resource 
tracking, block 300, is a process that discovers network information by examining the host 
network traffic. The analysis server looks for servers, protocols, and services and records 
them in the database. After being recorded, these sendees, protocols, etc. are tracked in the 

25 future and statistics about Ihem are gathered. This process is continuous, so new resources 
are discovered and old resources are deleted, generating alerts in either case. By listing this 
data, systems, protocols, and services in use at any given time are identified and a measure of 
how important they are to the host network is provided. 

Rules-based anomaly detection, block 350, includes using a series of mles to find 

30 anomalies on the network. At this point, anomalies are individual packets that match 
selection rules that define such things as illegal packets, unusual packets, and normal packets 
that are not sent to valid network resources (e.g., http requests to a server that does not 
service http). These packets are collected and added to the database in a protocol 
independent area called the anomaly pool. There are a series of normal rules that then 
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dependent on more than one possible value. If tiie abstracted threshold feils, the source data 
is examined and compared agamst the model mles once again. Since data is compared in the 
abstract layer to historical data, historical source data is necessary for full analysis. For each 
statistic, both generation I and generation n, each value is checked against the most recent 
5 prediction. If it fails, an alert is generated. New predictions for the next time interval (e.g., 
an hour) are calculated and stored. Generation n alerts are weighted heavier then generation 
n when creating alert prioritizations. Frequencies are checked by threshold predictions. Any 
remaining violations are added to the alert pool and submitted for adaptive classification. 

After threshold analysis is complete, alert correlation occurs, block 800. Alerts are 

10 correlated by type using statistical analysis of the types of alerts received. The correlation 
measures the number of alerts of various types and relates increases and decreases of alerts 
and their relationships to the percentage of the whole. The correlation is then performed for 
all alerts on the specific network. Related sensors are analyzed and alert relationships by 
source of alert (attacker) are noted. Attackers are recorded and tracked in each individual 

15 sensor. Upon completion of alert correlation, behavioral analysis for a particular sensor is 
complete. 

Global analysis, block 900, includes analysis of all available sensors as if they were 
on one large network. For example, if an IDS provider services one hundred networks or 
information systems, the global analysis may be performed for all sensors on these one 

20 hundred networks. Global analysis begins with the grouping of all alerts fi-om all sensors into 
a common alert pool. Where alerts are generated becomes irrelevant, only the alert types and 
sources are significant The common alert pool is then examined by alert type and by source, 
similar to the steps performed in the behavioral analysis for each sensor. This yields new 
prioritization values of both attackers and alert types. This new information may then be sent 

25 to the sensors to help prioritize alerts on a per sensor basis, block 950. 

More detailed information about certain exemplary embodiments of processes in 
blocks 300-950 of Figure 2 is shown in Figures 3-9 and described below. 

Referring now to Figure 3, process flow for an embodiment of resource tracking 300 
and an embodiment of rules-based anomaly detection 350 according to systems and methods 

30 of the present invention are shown. Initially, data is split by protocol and date in an RDB 
server, block 290. Periodically (e.g., hourly, bi-hourly), packets collected on a sensor are 
sent to the RDB server. These packets are split by protocol (e.g., ICMP, IP, TCP, and UDP) 
and inserted into the database. A filter may be applied at the time of import to selectively 
insert only certain packets, which is helpfiil in specific situations where it is not desirable to 
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allowed by the protocol specification or are not specified by the RTC. These packets are not 
normal to a network because they violate the protocol rules, but can still be routed on a 

network. 

Packet port violations are added at block 360. Packet violations are usually protocol 
5 violations, but are found by the IDS in a different manner. Unlike protocol violations, packet 
port violations are packets dropped off the network by hosts upon delivery. The packets are 
so malformed they are dropped completely, such as a layer specifying that layer 4 is TCP 
when the packet does not conform to TCP. At block 362, packets from envelope pairings are 
added. These are packets that could not have been generated or delivered to the host network 
10 and are very often the result of spoofing activity or an envelope specifying the same source 
and destination. Type code anomalies are also added, block 364. All ICMP packets have a 
type and a code that specifies what type of message the packet is carrying. Type code rules 
find illegal or strange type code combinations. 

At block 366, type of service (TOS) anomalies are added to the pool. The TOS field 
15 is a part of the IP protocol. Only certain values can be in the TOS field and any value outside 
of these specified values is suspect At block 368, TCP session indicator violations are 
added. TCP state indicators are bits within protocols that are used to indicate or store the 
information necessary to the protocol during normal use. Many protocol specifications only 
specify what the indicators are used for, but not how to deal with conflicting indicators or 
20 unused bits in the indicator space. 

Because broadcast traffic has no specific destination, broadcast traffic is examined 
separately and packets are added, as necessary, based on protocol models, at block 370. For 
instance, TCP to broadcast addresses is not supported and should not be found on a network. 
While blocks 356-370 relate to adding anomalous packets to the anomaly pool, block 
25 372 subtracts normal packets from the pool. To find normal packets lhat meet all 
specifications and are flse of violations but that are sent to or originating fmm the wrong 
resources, all raw packets are considered and those that originate properly and have come 
from proper destinations, as compared to the resource list from resource tracking, are 
removed. This removes all expected traffic fix)m the raw packets. The remaining packets are 
30 added to the anomaly pool for analysis. At block 374, synchronization is performed to ensure 
that all add (blocks 356-370) and subtract (block 372) rules are complete before continuing. 

Once again, some traffic added to the anomaly pool may be considered normal to a 
particular network, so a set of static deletion rules removes all matching traffic from tiie 
anomaly pool, block 376. At block 378, normal rules are appUed. Normal rules are code- 
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examination of signature alerts at block 505. All alerts generated for a specified time period 
(e.g., an hour, two hours, etc.) are analyzed for further correlations in an attempt to generate 
more useful alerts. Alerts are grouped by time and the number of alerts are counted for high 
activity periods, block 510. Extremely high numbers indicate further analysis is necessary. 
5 Active periods also indicate times a hacker may choose to work, which is often a detectable 
pattern that can be used to find hackers when they switch addresses or attack origms. 

Alerts are grouped by source, block 515, over time using historical data. This allows 
the IDS to find slow activity where the attacker is performing actions slowly in an attempt to 
stay hidden by not generating alerts quickly. Such things as slow port scans, slow 

10 vulnerability probes, and slow attacks may be found. The attackers' database from the add 
alerts routine already tracks individual sources, so this analysis focuses on slow activity. At 
block 520, packet flow correlation is examined, also known as backscatter. Here, protocol 
models are used to examine packet flow. Communication sessions may be examined and 
compared to normal models to find deviations. Spoofed traffic on the host network, as well 

15 as the residual traffic generated when the host network's addresses are externally spoofed, 
may be found. As shown in Figure 5, historical look-ups, 530, may performed for assistance 
as necessary among blocks 505-520 by the EDS. 

All historical alerts are grouped by source and counted to determine the degree of 
activity for any source using any high frequency alerts, block 525. Once the processes in 

20 blocks 510-525 are complete, alerts are grouped by source and appropriate STR changes are 
calculated for each alert or time pattern, block 535. Alert patterns warrant high SSV 
increases, while time patterns results in only small adjustments. STR changes are applied to 
the attack sources, block 540. This increases the strength of all sources generating like alerts 
and results in higher SSVs for sources using like alert pattems or like times. 

25 At block 545, technique discriminators are applied to STR. A static table contains a 

list of alerts and a number representing the skill level required to perform the type of attack 
that generates that alert. This is representative of the intelligence level of the hacker or, at the 
very least, the hacker's ability to know when to apply the appropriate hacking tool. Applying 
this niraiber as a modifier to STR allows SSVs to be affected by attacker skill in an attempt to 

30 highlight stealthy or intelligent hackers tidat are above average. This is known as a 
**techmque discriminator." The actual value of the technique discriminator is static and 
assigned by human analysts. The value may be used globally across multiple sensors and 
multiple protected networks. 
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detennine these counts fbr a specified tinte int^ (e g., a half-hour, an hour. Some da^ 
flow analysisUnecessarytodetorminc some generadonlsutisticsbec^r^cert^pto^ 
such as TCP. require multiple packers to determine if sessions are vahd or , us, attemptei 
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Many generation I statistics are gathered and stored, resulting in continuous data curves that 
have periodic (e.g., hourly, daily) data points. The statistics to be gathered are based on the 
protocols analyzed. For generation I statistics, because the data points do not have to be 
predictable, gathering as many statistics as possible is the primary goal. If predictable values 
5 are gathered, it is advantageous, but values that are predictable for one network may not be 
predictable for another. Generation I statistics are stored in a separate table. 

At block 610, generation n statistics are created from generation I statistics by finding 
relationships between data points. This is typically an operator (human) process, but is based 
on protocol and service data flow models. A standard set of generation II statistics may be 

10 generated by using the protocol models to find ratios and relationships. For example, 
comparing TCP packets with the SYN (synchronize/start) flag turned on in one direction 
versus TCP packets with the SYN-ACK (synchronize/acknowledge) flags turned on in the 
other direction. Generation n statistics use two or more generation I statistics to create more 
predictable values of related information by calculating and comparing ratios and differences 

15 over time on the network. Generation U statistics are stored in the thresholds table and 
calculated during threshold analysis (see Figure 7 and accompanying description below). 
The step shown at block 610 is the recording and storing of values to use as generation n 
statistics using analysis tools that assist in finding useful, predictable generation n statistics. 

At block 615, generation HI statistics are recorded and stored. Generation III 

20 statistics calculate events over time. These are called firequencies because they track the 
occurrence of events over time. Three main types of generation HI statistics are calculated: 
envelope pairings, service usage by resource, and external service usage. Over time, 
generation III statistics produce hi^y predictable curves if fhey are adapted as the network 
chemges. 

25 Envelope pairings are calculated by counting the number of times two systems 

exchange information. All envelopes are counted and grouped, block 620, by unique pairs of 
servers. For example, src to dst and dst to src are grouped the same. The number of unique 
conversations is recorded as a statistic. This shows which servers talk to which servers, and 
records the time of the events. At block 625, usage counts of service resources are made by 

30 examining the resource table and counting service usage per server. These values are 
recorded as a statistic and show the number of times each given resource is used on the 
protected network, including recording the time of the events. Service usage is calculated in 
an outbound direction, block 630, to profile what extemal services the protected network 
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^ TMs^videsfteexter„aIse,vi«.u^dand™»berof«m«*esen™»s«.»a.d«>d 

records the time of the events. . ■ -^^^^ 

AH co™.. calculaW in bloclcs 620-630 may be adjusW by a d— 
block 635 or amoved entirely a. a «ning mechanism for fl.e sensor. Such adjn^en, or 
I^..ybcpcrfo=ned.r.movesnclr,bin.as«afflc.ene,a^^=^^^ 
Z may dlLt ^ s.a«c in a. 1-oric da^base (beeanse Hs«>rie nnn*«s arc u«d » 
rti™.L,yadin.«.epr.dictions,snc.lar.eob3n.esmayca„seani.bal^^W^ 
period ottime jtno.remo™d). A. block 640. additional RIC changes ^J^'^ ^ 
again, titis is a mnir^ mechanism nscd .0 force a s.tic value onro an RIC . *P .-d 
cL ftom distorting values in *e ft.»e. All sUtistic^ values are now calorUUd and may 
b. compared as thresholds to the last time interval-s predictions, as sho»-n mFrgure 7. 

Figure 7 shov. process flow for an embodimen. of ti^shold analysis 700 accordmg 
,„ systems and methods of tire present hrvention. ^esbold analysis «-l.d« a c^ 
pjction alg^thm ti». traol. data poin« over time in order to make 
'values, -nueshold analysis may be used to track any statistical value. ^ as mm*«o 
packets of a given protocol, or data potats. .ch as tire fte,uency of a partrchr .« ^ a 
source Statistical value refers . ti»se values that come ^ ^^^^ 
Lr^le hu. no, Wted to. number ofpacke.^ number of UDP packets, of Suled 

elections who., an RST flag is detected, and otirers. 

^ytiom raw packet data. A datapointorigbates ^m apatt^n^e H^h.^.^ 
either automatically or by request ftom an operator of tire IDS. Da. pom. 
fectjy ftom raw data, but rati«r are calculaU=d ^m oorreUted events and rcp^ 

any repeated series o, events that occur in a pr^icable f^on (i.e a pattern). Some 
eUes inctode. but are no. limited to, tire m^ber o, times a partrcular server <m *e 
, 3 generates a specific aler. tire distinct number of times a worm ^^^^ 
Tetwo* and tire average fte^cy of web-based atiacks for a given server. Boar statis^cl 
valuesanddatapointsaretreatedintiresamemamrerbytirresholdanalysrs 

Ke<e.ringr.ow,oFigure7.«sprocessiteraUsoncefore.chtbreshol4 Apre^cti^ 

i. made fbr each collection interval. This prediction is ften checked agamst tire actual data^ 
0 :rmaygener..a,erts.The«^sho.disdrens,oredfor.t^use^<-S™-^^ 

^ratobeusedto calcuUteandadjust^rture thresholds. The lastpredrctionm^rs,^ 

^ fceshold table (mark^l -Wi*-") witi. an prevrous predrctior. (m^ 

.archival"). In flris m^mer. tire predicted dau curves are constantiy changed as the nehvo* 
changes. All ti^lds carr be cdcalated. for example, contimrously (H histonc- dati. 
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points), by time ranges (e.g., from 1 8:00 to 07:00), or as individual homs (e.g., from 06:00 to 
07:00) and each of these can be predicted. For each date-time range, there is a separate 
prediction. This allows for normal behaviors, such as outbound web usage increasing from 
08:00 to 18:00 on weekdays (a very normal behavior if user workstations are in the protected 
5 network), to be taken into account 

Alert correlation begins against the predicted threshold (calculated at the last anal3^is 
run) in block 702. The current statistic or data poiut being analyzed is compared to the last 
prediction, block 704. The prediction consists of four values: the actual data point, the 
predicted data point, the minor offset, and the major offset The data value is calculated 

10 using the new data received in the current data package (after all the statistics have been 
generated). The prediction of the current statistic/data point being analyzed is retrieved from 
the database along with the major and minor offsets. Adding the minor offset to the predicted 
value creates the upper limit and subtracting the minor offset from the predicted value creates 
the lower limit. This is done again with the major offset to create the major bracket These 

15 values create two brackets above and below the prediction that provide a buffer for the 
prediction to succeed. The higher the predictability of the given statistic/data point, the 
narrower the minor and major offsets and the smaller the brackets. 

A determination is made as to whether the current statistic or data point is outside the 
major bracket, block 706. ff the current statistic is outside the major bracket, this indicates a 

20 major threshold overflow, if greater than the upper major bracket, or major threshold 
underflow, if less than the lower major bracket ff the current statistic is outside the major 
bracket, an alert is added to the console, block 708, indicating the major threshold violation, 
but because the source of the problem is unknown, there are no changes made to the attacker 
database and the envelope is null. If the major bracket passes, tlie minor bracket is checked 

25 in the same way, block 710. Otherwise, the nunor bracket check is skipped, ff the current 
statistic is outside the minor bracket, an alert is added to the console, block 712, reporting the 
minor threshold violation. Once again, the source is unknown so the envelope is null. 

If either a nmjor or minor threshold violation occurs and an alert is added to the 
console, the next prediction is calculated at block 726, which is further described below, ff 

30 there are no major or minor threshold violations, meaning the last prediction made is correct, 
the RTC is increased at block 716. TTiis makes the prediction value more resistive to change 
as predictions become more accurate. An accurate prediction over a long period of time has a 
high RTC and refuses to change the value at all until flie RTC is lowered sufficiently (in 
block 748). In block 71 8, the STR is increased on a successful prediction by a percentage of 
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^ *o«.»rmpredic..bffi.y. For example, in . system wi.h analysis cv«y hour, fte smcan 
Z^r^l^ twenty-four poi„« rn a day. if .he nnnrber of snccesaM predrCo,. « a 
::Tsm1 Apreaio.aHli.yvalueisirr..ase.at«oo.720. lf*eKTCissu«c«.flylo. 
(as.aticvdne),toac*.alval.efromfl.esUtis«csmaybean=Mved.blockm^ 

A.block724.«reB.ndofmepredicSo„iso.lculated. Ws isnotftetrendof ft.da« 
^ « ..end of ^ prediction, winch is caUed the predicUhiU^ and n>e.s„res ^ 
abiUry of sys^m .» .rac. a specific da.a cnrve. This is «acW using two vahes. ^ 
^ bow n^a.^ conseo^ve «nres *e prediCon bas been cornet or n^ 
*o abiUty of *e cnrr^at sewings to predict *e daia. and dre p«cen«ge 
is correC overaU (i.e.. d.e long-tenn predic.biU.y of *e da.) Tbe con-^ 
Tf ^res. two values is *e predicabib^ of me da.a. If par^cular da. ponrts nev. s.abd.^^ 
«>ey can be dropped If they were previously predictable, an analysis roubne nray be rm,^ 
a^L nun^er of zones, d. .erg., percen^ges for dre drree «ers. me trnre ranges. ^ ^ 

Juple ^ of historic data (these para«e«rs wUl be be«er nnders«,od «> ^ 

Zssionbel„wbeginr^gwi«.hloc.726).B.chtin.easin^eva.iab^i3cta.ged 

are retroactively applied, and the resuhs data is scored based on rts accura^^ 

L actual data. The hi^st scoring set of paranreters can be instated on the '^J^^ 
Le:^ro«ineis,ypicanyonlymn When Short-term predic.abiH^.suns.ableft>rada,a 

curve with a bitfi long-term predictabiBty. 7n<! 

Beginning at block 726. a new prediction is calculated follcwmg e«be, blo^k 708. 
7.2 or 724. as shown in Figure 7. At block 726, a distrtburion over rime ia calculated ^ 
ml Fi^ a nmge is found providing *e values to use. Multiple predrcbons may be 
iTfor algl. data ^e breaking it into ranges of rime. Tbe minor and — 
^.vah«slcalcula.ed.giving.heuppera,.lowere«en..ofdrevalue^ ^ — 

5 to lower from the upper extent, d>e result is tbe range over wtacb dr. dari. curve 

range is avided into an e,ual number of .ones starring at dre low. e»^^ 

^^endingtotheupper extent Ibe number of ^ncmay vary, but is typrcally atleas. 10 «d 
^^20. Thelriburionof.heactnalprevioussamples.*om.hehisU,ricda«,ov.^ 

^nes is calculated, and a target percentage is selected. Tbe target 
,0 curve's p^tabOHy, or is 10%, for example, if there is no bistonc daU. Tbe bracket^ 
adjusted based di»iburion by dropping upper and lower .ones *at do no. co^ mo. 
to d» ^rge. percenmge of data, and rire zones are recalculated wi* ^ new extents. Thrs 
COTrinnes unril fl« target is achieved or «» process Ms. 
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At block 728, a tier one bracket is calculated. The tier one bracket is wide with a 
target of 10%. If the data conforms to the tier one bracket, block 730, (that is, 90% of the 
curve hits a single zone), the prediction is set for the median of the final extents. If tier one is 
achieved at the 10% target, then the new prediction is added to the database and the old 
5 prediction is archived, and a tier two bracket is calculated, block 732. If the data does not 
conform to the tier one bracket, then the prediction feils, block 742. The predictability trend 
values are adjusted downward, block 744. The number of consecutive successful predictions 
in a row is set to -1 to indicate the trend is one failed prediction. Long term predictability is 
recalculated. The STR is decreased, block 746, by a percentage of Ihe short-term 
10 predictability. The RTC is decreased, block 748, but only by a small amount Once all 
thresholds are checked and adapted to current network behaviors, alert correlation begins 
(Figure 8). 

If a tier two bracket is calculated, block 732 the tier two bracket is sUghtly smaller, for 
example, with a target of 6%. If the data conforms to the tier two bracket (that is, 94% of the 

15 curve hits a single zone), block 734, the prediction is set for the median of the final extents. 
If tier two can be achieved at the 6% target, the new prediction is added to the database and 
the old prediction is archived, ai^d a tier three bracket is calculated, block 736. At block 740, 
the tier one prediction is used if the tier two comparison fails (i.e., the data was not 
predictable enough to hit the 6% target) at block 734. 

20 The tier three bracket is very narrow and is targeted at, for example, 3%. A tier three 

bracket is calculated, block 736. If the data conforms to the tier three bracket (that is, 97% of 
the curve hits a single zone), block 738, the prediction is set for the median of the final 
extents. If tier three can be achieved at the 3% target, then the new prediction is added to the 
database and old prediction is archived. At block 740, the tier two prediction is used if the 

25 tier three comparison fails (i.e., die data was not predictable enough to hit the 3% target) at 
block 738. The tier three prediction is used at block 740 is the tier three comparison was 
successful. As the prediction is compared to each of the brackets, the prediction data value or 
point is the same, only the allowable offset changes. The prediction is stored as the value, the 
offsets, the target value, the zone size, number of zones, and so on (i.e., everything necessary 

30 to calculate ihe next prediction). Once all thresholds are checked and adapted to current 
network behaviors, alert correlation begins (Figure 8). 

Figure 8 shows process flow for an embodiment of alert correlation 800 according to 
systems and methods of the present invention. At fliis stage of behavioral analysis, all raw 
data has been analyzed. Alert correlation examines all alerts for the sensor to find 

25 
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^g^le pa^ms «u. n>ay b= used «, clasriiy aler.,. Or^ oc^mo. has W 

performed for <^ sensor. c«re,ation is performed across all sensor on «» prote«ed 
rretworlc Alc«s ... grouped by source, block 805. so flrey may be easit- exanur^d. Atb^ook 
810 if taK>»n sequences of alerts are found, additional alerts are generated, blook 845^» 
5 iurtor classic the attact For example, eleven nS:CMD.EXE alerts, three nS:UNICODE 
alerts ^ two nS:WIC0DE2 alerts aU torn the same source in a specific order ^ a Nnnda 
worm' atteet Tie addiliond alert generated is WOEM:MMDA. 

The sources «e grouped by alert type, block S15. This shows a count of drstmct 
sources for each type of al«t A, block 820. it is determined whether there is a high number 
10 ofdistinctsouroes. The hitfrer the distinct count of sources for each aler, the more acttve the 
event (associa.«l with to alert) is on the network. This infonnation may also be used to 
adjust the technique discriminators used in block 545 (see Figure 5) and to note new a«.* 
Jnds Iffhe..is.highnumberofdistinctsources.analertUadded.block845. Anynew 
sequences fo«.d added to the relational daubase. block 840, and ti« process reruns to 
,5 blook815. FoUowing analysis at block 820. alert correlation proceeds to block 825. netwo* 
oorrelation. N«wo4 conelation looks for tire same alerts on multiple sensors .m 4e 
protected network. Alert, torn reUted sensors are pooled into a single alert pool Oogtcally. 

but not necessarily pliysically). j -m. 

At block 830. an examination for matches from sensor to sensor is performed. THe 
20 direction of each alert is loio^vn. An alert moving from an external sensor to an mtem^ 
sensor should first hit fl.e external system ti^en^e internal system. When an alert on an 
external sensor has amatching alert of the same source (within an appropriate time frame) on 
an internal sensor, it indicates «.e network has been penetrated (vice versa for an outgomg 
attack) If no matches axe found, alert correlation ends. If matches are found, the raw 
25 packetsaresearched,block835.forareplyindicatingtheattackworked. If so, the alerts are 
escalated to a hi^ priori^. Block 845 is a standard add alerts, and because the sources are 

known, tiie attacker's SIR is increased. 

A global andysismay he performed including analysis of aU available sensors across 
™^,e netwo-ks. Figure 9 shows process flow for an embodiment of ^obal ana^^ 900 
30 according to systems a»i methods of the present invention. Global analysis ,s desrgned to 
anahze attack sources ftom the mtemet across the whole customer profile. By perfomn^ 
«s analysis, scanning or attack soun=es can be identified across subnets or across the 
in,^. Additionally, if coordinated atincks are occurring and are being performed agamst 
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multiple sites, they can be found using global analysis. As the number of data sources 
increases, the quality of this analysis improves. 

Global analysis may be used to analyze thousands or even himdreds of thousands of 
sources across the Internet at a central site in an attempt to find sources of suspicious activity. 
5 Attackers can perform vulnerability scans on many targets at once, and often do so in an 
attempt to hide the attack. Listead of scanning one target for 500 different vulnerabilities at 
once, they may scan 200 sites for the same vulnerabilities. The attacker's computer may be 
busy twenty-four hours a day, but each site receives a tiny portion of scan each day and 
therefore manages to stay under the rate that is detectable by traditional knowledge-based 
10 systems. 

Referring now to the exemplary global analysis shown in Figure 9, alerts are counted 
by alert type for soxirces outside the protected networks of the sensors at block 905. This 
yields alert frequencies by type, which indicate attack- methods being used. At block 910, 
alert counts are added to the RDB. Alerts are then counted by source and priority, block 915, 
15 showing which attackers are most active on the global network. At block 920, attacker 
information from block 915 is added to the RDB. Global alerts are weighted by priority, 
block 925. Higher priority alerts are given additional counts in both alert type tables and 
attacker tables. 

At block 930, in a first table, a set of alerts with a weighted count representing the 
20 type of alerts on the global network is created. In a second table, global attack sources are 
also weighted by priority, with both tables being sorted from high to low. Sensors are 
adjusted, block 935. Attacker source strengths are raised in sensor attacker databases so alert 
SSVs from the same sources are automatically higher. Alert type default priorities may be 
adjusted, block 940, based on the global alert frequencies. These can be pushed to either 
25 default priorities of alerts or to the static technique discrimiaators (preferred). Once global 
analysis is complete, the results are sent to each sensor for each protected network analyzed 
as part of the global analysis (see block 950, Figure 2). 

Optionally, systems and methods of intrusion detection according to the present 
invention may include an alert release system to assist operators of the IDS. Process flow for 
30 an embodiment of an alert release system is shown in Figure 10. This system is designed to 
reduce the number of alerts an operator of the EDS has to view while maintaining a larger 
number of alerts for correlation and analysis. Alerts are created in a generational system, and 
higher gmeration alerts are shown on the operator on the console. Alerts are selected for 
display by a series of rules that define flie alerts that should be displayed. The series of rules 
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^ ^^ally Mgh level election palmate., bu, may be «^ « scleettvely *ow or .g-oxe 
are generally mgn lev . . t ,™ote imoitag all alerts fixim a 

alelB based on very specific cntena. such as, for example, ign g 

particular source or ignoring aU alerts in a particular time wmdow. 

Tte series of rules may be appUed in a specific order such «». one. a ™gle nrl. r 
app,iedL«ngto-indica«on«..«.e alert Should no.be displayed.ft.rther«les„olon^ 

::^t^PpL«,«».alortand«. alertisnordisplayed. -«*<«*--'"2 
on a certain (possibly, more reliable) rule is wei^ted heavier than fte mdica^on based on 

,5 "'"Is an example of *e «mc.onaU.y alert release sys.em, consider *e Ni^da 

Bive^lusedlde»nnine«aw.ba.«.isacn.anyal«mdawma«ac,.Thefi« 

^^TTno. released a. console, ins^d a new alert called Nimda is -^J^ 
rair.released.^econsole. ^s preserves ihe record *a. «ve recurred bu. 

*tf ftev were all ftom «» same somce (as ,»ilh a Nimda worm a«ack). 
30 ^1. release sysrem also allows sources and services » be rename.^^^ 

name resolndon fcr alerts so ta. servers (including in«mal servers where no pubta DNS^ 
::ir-rbLnsmed. A«.bleres„lvesaddresses«names. For example, before a^^ 

rtl^» l console. *e address^ are cbang^ .o nreaningM names s„=^ - 
::^^op«.»rof*elDSse.s*enamem«.erU.an«.eaddress.which.sbeneficu> 
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because the human operator is more likely to immediately recognize the name. Address 
information is maintained in the alert for the operator's use and reference. 

Additionally, alerts may be renamed after alerts have been generated and imported 
from a signature-based server using this alert release system. Previously, all importers 
5 maintained a list of alert names that needed to changed at the time of import, each of which 
were unique to the signature-based system generating the alerts. The alert release system 
allows a single list of alert renames to be used to change all alerts of a particular name to 
another name to assure unified data names. 

Referring now to Figure 10, where an exemplary process flow for an embodiment of 

10 an alert release system according to systems and methods of the present invention is shown, 
at block 1005, alerts are added to a database at generation one. IP addresses are resolved to 
names, block 1010, and alerts are renamed, block 1015. Alerts are reclassified, block 1020, 
and a series of selection rules are used to select alerts tiiat are to be displayed on the console, 
block 1025, for viewing by the operator of the IDS. The alerts selected for display are 

15 assigned generation two status, block 1030, and stored in a generation two alert database. 

The foregoing description of exemplary embodiments according to systems and 
methods of the invention has been presented only for tiie purposes of illustration and 
description and is not intended to be exhaustive or to limit the invention to the precise forms 
disclosed. Many modifications and variations are possible in light of the above teaching. 

20 The embodiments were chosen and described in order to explaiu the principles of the 

invention and thek practical appUcation so as to enable others skilled in the art to utilize the 
invention and various embodiments and with various modifications as are suited to the 
particular use contemplated. Altemative embodiments will become apparent to those skilled 
in the art to which the present invention pertains v/ithout departing from its spirit and scope. 
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Claims 

What is claimed is: 

1. A method for detecting network intrusion attempts associated with network 
objects on a coimnumcatlons network, the method comprising: 

collecting normal traffic behavior associated with network objects on the 
networkonacontinuingbasis to establishhistoricaldataregarding traffic acrossfl^ene^^^ 

monitoring network traffic associated wi* network objects on Ihe network to 

detect anomalies; 

analyzmg the anomalies using the historical data; 

generating alerts identi^ng possible intmsion attempts based on analysis of 

the anomalies; and ^ ^ 

updating the historical data based on the anomalies, the alerts, and network 

traffic. 

2 The meflioa of date 1. wherem monitoring nowork traffic to detect 
^es comprtaes =»>mtort.g traffic for to,™ ^ and series of *,1es *at 
indicate signature attacks. 

3 Tte method of claim 1. wherein monitoring network naffic to detect 
^malies fWher comprises appWng a series of mles to identift, anomalons packets arKi 
adding the momalous packets to an anomaly pool. 

4 The method of claim 3, wherein analyzmg the anomaUes using Ihe historical 
data comprises analyzing packets in the anomaly pool independently of any of 4e serres of 
rules that identified the packet for addition to the anomaly pool. 

5 The method of claim 1, wherein analyzing the anomalies using the historical 
data comprises conducting a threshold analysis to determine whether a statistic or data pomt 
is wilhin threshold values. 

6 The meflrod of claim 1. wherein generating alerts identiftrtng possible 
^tempts comprises adding ale-s «> an alert pool and releasing aleris to a console fcr 

viewing by an operator. 
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7. The method of claim 1, wherein generating alerts identifying possible 
intrusion attempts comprises: 

adding a first set of alerts to an alert pool; 

resolving internet protocol addresses associated with each alert in the alert 
pool with a name and renaming each alert in the alert pool with a name recognizable by an 
operator; 

applying a set of rules to select alerts from the alert pool to be displayed on a 
console for viewmg by the operator, the rules comprising high level selection parameters that 
have been previously defined; and 

releasing the selected alerts by name to the console for viewing by the 

operator. 

8. The method of claim 1, further comprising performing the method across a 
pluraUty of nehvorks and compiling results in a global database. 

9. The method of clann 8, further comprising updating the historical data based 
on results in the global database. 

10. The method of claim 1, wherein updating the historical data includes storing 
sources of possible intrusion attempts in a database with an indicator of their hostility to the 
network and removing a source of possible intrusion attempts from the database when a value 
of the indicator is sufficiently low due to a period of inactivity of the source. 

11. A computer storage medium storing a computer program which, when 
executed by a computer-controlled apparatus, causes the computer-controlled apparatus to 
perform the method of claim 1 . 

12. A computer-controlled apparatus operative for implementing the method of 
claim 1. 

13. A method for detecting network intrasion attempts associated with- network 
objects on a commtmicatioiis network, the method comprising: 
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coUecting normal traffic behavior associated with network objects on the 
networkonacontinuingbasisto establishhistorical dataregarding traffic across thenetwo^^^ 

xnomtoring network traffic associated with network objecte on the network to 

detect anomalies comprising: - ^ ^ 

looking for known strings and series of bytes that indicate signature 

attacks; and , 
applying a series of rales to identify anomalous packets and adding the 

anomalous packets to an anomaly pool; 

analyzing flie anomalies using the historical dat^ 

generating alerts identifying possible intnxsion attempts based on analysis of 

the anomalies; and ^.^^^ 
updating the historical data based on tiie anomaUes, the alerts, and netwoA 

traffic. 

14. The method of claim 13. wherein analyzing &e anomaUes using the historical 

data comprises: • 

analyzmg packets in the anomaly pool independently of any of the senes of 
rules lhatidentified the packet for addition to the anomaly pool; and 

conducting a Ihieshold analysis to determine whether a statistic or data pomt is 
within fhieshold values. 

15. -me mefliod of claim 13. wherein generating alerts identifying possible 

intrusion attempts comprises: 

adding a first set of alerts to an alert pool; 

resolving internet protocol addresses associated with each alert in tiie alert 
pool with a name and renaming each alert in the alert pool with a name recognizable by an 

applying a set of rules to select alerts from the alertpool to be displayed on a 
console for viewingby^e operator, the rules comprisinghighlevelselectionparamete. 

have been previously defined; and 

releasing the selected alerts by name to the console for viewmg by tire 



operator. 



16. The method of claim 13, fiirlher comprising: 
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performing the method across a plurality of networks; 
compiling results in a global database; and 

updating the historical data based on results in the global database. 

17. The method of claim 13, wherein updating the historical data includes storing 
sources of possible intrusion attempts in a database with an indicator of their hostility to the 
network and removing a source of possible intrusion attempts from the database when a value 
of the indicator is sufficiently low due to a period of inactivity of the source. 

18. A computer storage medium storing a computer program which, when 
executed by a computer-controlled apparatus, causes the computer-controlled apparatus to 
perform the method of claim 13.. 

19. A computer-controlled apparatus operative for implementing the method of 
claim 13. 

20. An intmsion detection system for detecting network intrusion attempts 
associated v^th network objects on a communications network, the system comprising: 

a sensor connected to the network for monitoring network traffic associated 
with network objects on the network comprising: 

a knowledge-based component for examining network trajffic for 
known strings and series of bytes that indicate signature attacks; and 

a packet logger for reading packets in network traffic, classifying 
packets by protocols, and creating packages of compressed packets; 

a server connected to the sensor that accepts real-time alerts for possible 
signature attacks and a converter for converting alerts from native signature format to a 
unified format for storage in at least one relational database; 

an analysis server that receives compressed packets from the sensor at periodic 
intervals, wherein the analysis server conducts a behavioral analysis of the data received from 
the sensor; and 

the at least one relational database, which stores raw packet data, behavioral 
data, and index data. 
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21. Tl,esystemofclaim20.furtheroomprisingapluraUtyofsensorscoTmected 
the network. 

22. THe system of claim 20. further comprising two or more virtual private 
network tunnels connecting Ihe sensor to the network. 
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